Moleclar property modeling using ranking

ABSTRACT

Methods and articles of manufacture for modeling molecular properties using data regarding the partial orderings of compound properties, or by considering measurements of compound properties in terms of partial orderings are disclosed. One embodiment provides for constructing such partial orderings from data that is not already in an ordered form by processing training data to produce a partial ordering of the compounds with respect to a property of interest. Another embodiment of the invention may process the modified training data to construct a model that predicts the property of interest for arbitrary compounds.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional patent applicationSer. No. 60/584,819, filed Jun. 29, 2004, and to U.S. Provisional patentapplication Ser. No. 60/584,820, filed Jun. 29, 2004, both of which areincorporated by reference herein in their entirety.

This application is also related to the following: (1) U.S. Pat. No.6,571,226, Issued May 23, 2003, (2) U.S. patent application Ser. No.11/074,587, filed on Mar. 8, 2005, (3) U.S. patent application Ser. No.10/449,948, filed on May 30, 2003; (4) U.S. patent application Ser. No.10/452,481, filed on May 30, 2003, and (5) U.S. patent application Ser.No. ______, filed on even date herewith entitled “Estimating theAccuracy of Molecular Properties Models and Predictions”. Each of theaforementioned patent and patent applications are incorporated byreference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to machinelearning techniques and, more particularly, to a method, article ofmanufacture and apparatus for modeling molecular properties using rankeddata and ranking algorithms.

2. Description of the Related Art

Many industries use machine learning techniques to construct predictivemodels of relevant phenomena. For example, machine learning applicationshave been developed to detect fraudulent credit card transactions,predict creditworthiness, or recognize words spoken by an individual.Machine learning techniques have also been applied to create predictivemodels of chemical and biological systems. Generally, machine learningtechniques are used to construct a software application that improvesits ability to perform a task as it analyzes more data related to thetask. Often, the task is to predict an unknown attribute or quantityfrom known information (e.g., credit risk predictions based on priorlending history and payment performance), or to classify an object asbelonging to a particular group (e.g., speech recognition software thatclassifies speech into individual words). Typically, a machine learningapplication improves its performance using a set of training examples.Each training example may include an example of an object, along with avalue for the otherwise unknown classification of the object. Byprocessing a set of training examples that include both an object and aclassification for the object, the model “learns” what attributes orcharacteristics of the object are associated with a particularclassification. This “learning” may then be used to predict theattribute or to predict a classification for other objects. For example,speech recognition software may be trained by having a user recite apre-selected paragraph of text. By examining the attributes of therecited text, the software learns to recognize the words spoken by theindividual speaker.

In the fields of bioinformatics and computational chemistry, machinelearning applications have been used to develop models of variousmolecular properties. Oftentimes, such models are built in an attempt topredict whether a particular molecule will exhibit the property beingmodeled. For example, models may be developed to predict biologicalproperties such as pharmacokinetic or pharmacodynamic properties,physiological or pharmacological activity, toxicity or selectivity.Other examples include models that predict chemical properties such asreactivity, binding affinity, or properties of specific atoms or bondsin a molecule, e.g. bond stability. Similarly, models may be developedthat predict physical properties such as the melting point or solubilityof a substance. Further, molecular models may also be developed thatpredict properties useful in physics-based simulations such asforce-field parameters or the free energy states of different possibleconformations of a molecule.

The training examples used to train a molecular properties model eachtypically include a description for a molecule (e.g., the atoms in aparticular molecule along with the bonds between them) and dataregarding the property of interest for the molecule. Collectively, thetraining examples are commonly referred to as a “training set” or as“training data.” Data regarding the property of interest typically takesone of two forms: (i) a value from a continuous range (e.g., thesolubility of a molecule at a solute temperature), or (ii) a labelasserting presence or absence of the property of interest relative tothe molecule included in the training example. In either case, thetraining examples measure the property of interest relative only to themolecule included in a particular training example.

Using training data in either form has often, however, proved to beineffective in training molecular properties models with a useful degreeof predictive power. This may occur due to problems with the quality ofthe training data. First, consider a scenario where the data is anumerical value representing a measurement of the property of interestover a continuous range. The measurement values available for aparticular molecule frequently differ depending on the data source. Forexample, measurements obtained from one lab or using one experimentalprotocol may consistently assign higher values for a property ofinterest to a particular molecule than others. These differences oftenlead to inconsistent values for the property of interest being reportedfor the same molecule. Additionally, even measurements obtained under“identical” experimental conditions may have enough experimentaluncertainty or noise that it becomes unreasonable to assign a precisenumerical value to the property of interest. One reasonable observationunder these circumstances may be that if the difference in, or relativemagnitude of, measurements reported for two different molecules is largeenough, then one molecule may be said to have “more” of the propertythan the other.

Measurements for a set of molecules may be either relative or absolute.For example, this is commonly encountered in molecular modelingcalculations where the ranking of molecules based on the calculation ofabsolute binding energies can be less accurate than the ranking ofcompounds based on relative calculated binding energies.

Training examples that use a label asserting the presence or absence ofthe property of interest have also proven to be of limited value intraining a molecular properties model. Oftentimes, such data has a largebias in that the data is predominantly of one label. (e.g., nearly allof the molecules are “inactive” for the property of interest). In thiscase, it is easy to obtain a model with high accuracy; the model simplypredicts the predominant label (e.g., always predict that a moleculewill not have the property of interest). This model, however, is notparticularly useful, as it makes the same prediction for every molecule.

Generally, models built from data will not predict the property ofinterest with perfect accuracy for all molecules, and there will be someerrors. For binary valued data (i.e. training examples that use a labelasserting the presence or absence of a property) these errors consist offalse positives (i.e. molecules falsely predicted to have the propertyof interest), or false negatives (i.e. molecules falsely predicted tonot have the property of interest). These types of errors have differentcosts, (e.g., in a diamond mine it is far more expensive to falselypredict that a diamond is dirt than it is to predict that dirt is adiamond). In biological and pharmaceutical applications, however, it canbe very difficult to assign relative values to false positives and falsenegatives and so it becomes very difficult to trade them off.

As these examples illustrate, it is often easier (and more accurate) toconsider the ordering of two molecules relative to a certain propertythan it is to assert an absolute value for the property for a singlemolecule. Existing molecular property modeling techniques, however, arenot capable of using such ordering information, nor are they capable ofdealing with bias in the data or of constructing reasonable modelswithout knowing the optimal trade-off between false positives and falsenegatives. Accordingly, there is a need for improved methods andapparatus for modeling molecular properties.

SUMMARY OF THE INVENTION

Embodiments of the invention provide methods, apparatus, and articles ofmanufacture for training a molecular properties model. Specifically,embodiments of the invention provide novel techniques for trainingmolecular properties models that order (or rank) sets of molecules withrespect to a property of interest. Embodiments of the invention providenovel techniques for generating ranked training data used to train amolecular properties model. Further, embodiments of the inventionprovide novel techniques for training molecular properties models basedon data provided in a ranked form. Further, embodiments of the presentinvention provide novel techniques for training molecular propertiesmodels that order sets of molecules relative to a property of interestbased on data that is not provided in a ranked form. Further,embodiments of the present invention provide novel techniques fordealing with the bias in training data and for constructing an accuratemodel despite not knowing the trade-off between false positives andfalse negatives a priori. One embodiment of the invention provides amethod for generating a pseudo-partial ordering of ranked pairs ofmolecules, used to train a molecular properties model. The methodgenerally includes obtaining a set of property measurements for aplurality of molecules, wherein each measurement assigns a value for aproperty of interest relative to a single molecule, selecting pairs ofmolecules from the plurality, wherein a first and second molecule, in apair of molecules, are ordered relative to one another and the propertyof interest, and combining the selected pairs of molecules to form thepseudo-partial ordering of ranked pairs.

Another embodiment provides a method for training a molecular propertiesmodel that includes obtaining a pseudo-partial ordering of ranked pairs,wherein each ranked pair includes at least a representation of a firstand second molecule, ordered relative to one another and a property ofinterest, and generating a representation of the molecules included inthe pseudo partial ordering of ranked pairs that is appropriate for aselected machine learning algorithm, wherein the pseudo partial orderingof ranked pairs is provided to the selected machine learning algorithm,and wherein executing the selected machine learning algorithm, using theranked pairs, trains a molecular properties model configured to generatea prediction regarding additional molecules supplied to the model.

Another embodiment provides a method for training a molecular propertiesmodel that generally includes, selecting at least two molecules toinclude in a ranked ordering of molecules, wherein the ranked orderingof molecules orders each molecule in the ranked ordering, relative toone another and relative to a property of interest, providing the rankedordering to a selected machine learning algorithm, and executing themachine learning algorithm to generate a trained molecular propertiesmodel.

Another embodiment provides a computer-readable medium containing anexecutable component that, when executed by a processor, performsoperations that generally include receiving, in a computer readableform, a set of property measurements for a plurality of molecules,wherein each measurement provides a value for a property of interestrelative to a single molecule, selecting pairs of molecules, from theplurality, wherein a first and second molecule, in a pair of molecules,are ordered relative to one another and the property of interest, andcombining the selected pairs of molecules to form the pseudo-partialordering of ranked pairs.

Another embodiment provides a computer-readable medium containing anexecutable component that, when executed by a processor, performsoperations that generally include, selecting at least two molecules toinclude in a ranked ordering of molecules, wherein the ranked orderingof molecules orders each molecule in the ranked ordering, relative toone another and relative to a property of interest. The operationsgenerally further include providing the ranked ordering to a selectedmachine learning algorithm, and executing the machine learning algorithmto generate a trained molecular properties model.

Another embodiment provides a method for evaluating a prediction about amolecule, generated using a computer-implemented molecular propertiesmodel. The method generally includes receiving the prediction for atleast a test molecule generated by the molecular properties model,wherein the molecular properties model is trained using a set oftraining data, and wherein the training data comprises a pseudo-partialordering of molecules. In one embodiment, the molecular properties modelmay be trained by (i) obtaining a set of property measurements for aplurality of molecules, wherein each measurement provides a value for aproperty of interest relative to a single molecule, (ii) selecting atleast two molecules to include in the pseudo partial ordering, whereinthe pseudo partial ordering of molecules orders each therein, relativeto one another and relative to a property of interest, and (iii)providing the pseudo partial ordering to a selected machine learningalgorithm, wherein the selected machine learning algorithm executedusing the training data generates the molecular properties model. Themethod generally further includes determining the accuracy of theprediction for the test molecule by performing experimentation.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features, advantages andobjects of the present invention are attained and can be understood indetail, a more particular description of the invention, brieflysummarized above, may be had by reference to the embodiments illustratedby the appended drawings. These drawings, however, illustrate typicalembodiments of the invention and are not meant to be limiting of itsscope, for the invention may admit to other equally effectiveembodiments.

FIG. 1 illustrates a view of a computing environment used to construct amolecular properties model, according to one embodiment of theinvention.

FIG. 2 illustrates exemplary measurements of a property of interestreported for a set of molecules that may be used to construct apseudo-partial ordering, according to one embodiment of the invention.

FIG. 3 illustrates multiple sets of molecules and assigned activityvalues that may be used to construct a pseudo-partial ordering,according to one embodiment of the invention.

FIG. 4 illustrates a method for constructing a pseudo-partial orderingfrom a set of molecules, according to one embodiment of the invention.

FIG. 5 illustrates a method for training a molecular properties modelfrom a pseudo-partial ordering of ranked pairs, according to oneembodiment of the invention.

FIG. 6 illustrates a block diagram of data flow through a molecularproperties model trained to generate predictions about an arbitrarymolecule, according to one embodiment of the invention.

DETAILED DESCRIPTION

Embodiments of the invention provide novel techniques for modelingmolecular properties. Specifically, embodiments of the invention providenovel techniques for training molecular properties models that ordersets of molecules relative to a property of interest. Embodiments of theinvention generally train a molecular properties model in one of fourways:

-   -   (i) Embodiments of the invention provide novel techniques for        generating ranked training data used to train a molecular        properties model. Particular embodiments of the invention may be        used to generate ranked data from data that is not provided in a        ranked form.    -   (ii) Embodiments of the invention provide techniques that train        a molecular properties model using training examples based on        ranked data. Embodiments of the invention generate training        examples based on ranked data that may be used by a learning        algorithm that is not configured to process ranked data.    -   (iii) Embodiments of the invention provide novel techniques for        training molecular properties models that order sets of        molecules relative to a property of interest. Particular        embodiments of the invention may be used to train a molecular        properties model using training data that is not provided in a        ranked form, without explicitly generating data in a ranked        form. Particular embodiments of the invention may be used to        train a molecular properties model by (approximately) minimizing        a function of the order assigned to a set of molecules.    -   (iv) Embodiments of the invention provide novel techniques for        training molecular properties models that achieve trade-offs        between false negatives and false positives despite not knowing        the ideal trade-off a priori.

Although the subsequent discussion describes the invention in terms ofrank ordering multiple molecules with respect to a property of interest,the invention is not limited to these kinds of molecular properties. Forexample, embodiments of the invention may train a molecular propertiesmodel to rank order different 3-dimensional conformations of a singlemolecule. Further, embodiments of the invention may train a molecularproperties model to rank order different atoms or bonds in a givenmolecule with respect to a property of interest (e.g. the pKa or partialcharge of a selected atom or bond). Those skilled in the art willobserve how the following discussion may be applied in these cases.

An Exemplary Computing Environment

Embodiments of the invention may be implemented as computer softwareproducts for use with computer systems like the one illustrated inFIG. 1. Such programs may be contained on a variety of signal-bearingmedia. Examples of signal-bearing media include (i) informationpermanently stored on non-writable storage media (e.g., a CD or DVDdisk); (ii) alterable information stored on writable storage media(e.g., floppy disks within a diskette drive or hard-disk drive); and(iii) information conveyed to a computer by a communications network,including wireless communications. The latter embodiment specificallyincludes information made available on the Internet and other networks.Such signal-bearing media, when carrying computer-readable instructionsthat implement the methods of the invention, represent embodiments ofthe invention.

FIG. 1 illustrates an exemplary computing environment 100. Network 104connects computer system 102 and computer systems, 106 _(1-N). In oneembodiment, computer 102 comprises a server computer system configuredto respond to the requests of systems 106 _(1-N) acting as clients.Illustratively, computer systems 102 generally include a centralprocessing unit (CPU) 110 connected via a bus 112 to memory 116, storage114, and network interface 104, and the like. Computer systems 102 and106 _(1-N) also typically include input/output devices such as a mouse,keyboard, and monitor, and may include other specialized hardware Memory116 includes machine learning application 120 and PPO application 118.

Embodiments of the invention may be implemented using any availablecomputer system and adaptations are contemplated for both known andlater developed computing platforms and hardware. Accordingly, themethods described below may be carried out by software applicationsconfigured to execute on computer systems ranging from single-userworkstations, client server networks, large distributed systemsemploying peer-to-peer techniques, or clustered grid systems. In oneembodiment, computer system 102 and computer systems 106 _(1-N) may beconnected to form a high-speed computing cluster such as a Beowulfcluster, or other clustered configuration. Those skilled in the art willrecognize that a Beowulf cluster is a method for creating ahigh-performance computing environment by connecting inexpensivepersonal computer systems over high-speed network paths. In such anembodiment, computer system 102 may comprise a master computer used tocontrol and direct the scheduling and processing activity of computersystems 106 _(1-N).

Further, the computer systems used to practice the methods of thepresent invention may be geographically dispersed across local ornational boundaries using network 104. Moreover, predictions generatedfor a test molecule at one location may be transported to otherlocations using well known data storage and transmission techniques, andpredictions may be verified experimentally at the other locations. Forexample, a computer system may be located in one country and configuredto generate predictions about the property of interest for a selectedgroup of molecules, this data may then be transported (or transmitted)to another location, or even another country, where it may be thesubject of further investigation e.g., laboratory confirmation of theprediction or further computer-based simulations.

Creating Ranked Training Data

Rather than use training examples that provide a measurement for aselected property of interest relative to a single molecule, embodimentsof the invention use training examples based on a relative measurementbetween two or more molecules. The term “ranked data” refers to sets ofmolecules wherein the measurement for the property of interest for onemolecule is deemed to be greater (or lesser) than the activity of theother molecules in the set. For example consider the set of twomolecules {A, B}, if molecule A has a reported measurement value of 85,and molecule has a reported measurement value of 70, then molecule A issaid to be ranked greater than molecule B. This is represented herein bythe inequality (A>B), or, for short, just the ranked pair: (A, B).Although described herein using ranked pairs of molecules, those skilledin the art will readily recognize that the techniques disclosed hereinmay readily be extended to a vector ranking that includes an arbitrarynumber of molecules, ranked relative to one another (e.g., the rankingvector <a, b, c, d> wherein the ranking of one molecule is greater thanits right neighbor, and lower then its left neighbor, where one exists).

For example, data taken from Table 1 of the Journal of MedicinalChemistry, volume 48, pages 3118-3121, shows that compound 2a binds toChk1 kinase with an affinity of 3 nanomolar, whereas compound 2c bindswith an affinity of 10 nanomolar. Thus, the ranked pair (2a, 2c) may beused to represent a ranking of compounds 2a and 2c relative to thisproperty of interest.

As noted above, embodiments of the invention may be used to modelmolecular properties that correspond to properties of atoms or bonds ofa single molecule, or to alternative representations or conformations ofa molecule. For example, embodiments of the invention may be used tomodel an ordering of the possible three dimensional conformations of amolecule. Here conformation A for a molecule is ranked higher thanconformation if conformation A is more likely in some environment (e.g.a particular solvent). Thus, similar to embodiments used to processordered pairs of molecules it may likewise consider ordered pairs ofthree dimensional conformations of a given molecule.

In addition, property measurements may be related to atoms or bonds in amolecule. For example, the invention may be applied to construct a modelof the pKa of each atom in a molecule; in this case the model will rankthe atoms according to their pKa. Thus, the ranked pair (A, B) mayrepresent a ranking of different atoms of a single molecule, relative totheir pKa.

Using property measurements available for a set of molecules, a“pseudo-partial order” (“PPO”) is constructed. A pseudo-partial order ofmolecules is constructed from individual pairs of molecules, accordingto the available measurements and selection criteria. A partial order(represented using the symbol “≦”) is defined mathematically as arelation on a set with the properties of reflexivity i.e., antisymmetryi.e. (A≦B),(B≦A)

A=B and transitivity i.e. (A≦B),(B≦C)

(A≦C). A “pseudo-partial order” (“PPO”) is defined herein as therelation on a set that can be viewed as a partial order for whichantisymmetry does not hold and for which transitivity is does not hold.A PPO can be viewed as a partial order that has been corrupted by noise,or had errors introduced.

A PPO is partial because not all possible ranked pairs from the set ofmolecules are necessarily included. For example, consider molecules A,B, and C. If a PPO of these three molecules (relative to a property ofinterest) includes the following two ranked pairs: (A, B), and (A, C);it remains unknown whether the correct full ordering is (A, B, C), or(A, C, B).

Anti-symmetry does not hold for a PPO as the pairs (A, B) and (B, A) mayboth be contained in the PPO because one of the relationships isinferred from noisy or misleading data. Transitivity does notnecessarily hold for a PPO as inconsistencies in experimental resultsmay not imply transitivity for a given molecular property.

Accordingly, as used herein, a “pseudo-partial order” (or PPO) includesa set of ranked pairs. For example, the above set {(A, B), (A, C)} is anexample of a PPO. The ranked pairs included in a PPO may be inconsistentand include both (A, B) and (B, A) as ranked pairs. Further, a PPO mayinclude the same ranked pair more than once, and may not be transitiveacross ranked pairs. A PPO may be considered as a partial ordercorrupted by noise. Noise-tolerant learning algorithms may then induce amodel that assigns a partial order to a set of molecules.

The elements of a PPO may be associated with weights to create aweighted PPO. The meaning of these weights can vary, but oneinterpretation is that the weights correspond to a measure of theconfidence in the correctness of the given element i.e. the pair (A,B)may be assigned the weight 1.2, while the pair (B,A) may be assigned theweight 4.5, the interpretation being that the pair (B,A) is more likelyto be the correct ordering of the two molecules included in the pair.

Those skilled in the art will recognize that PPOs may be represented inmany ways. For example a PPO may be represented as a set of orderedtuples (A,B,C,D) wherein molecules in the ordered tuple are consideredto be ranked higher (or lower) than molecules that succeed them in theordered tuple. This set of ordered tuples can contain inconsistenttuples wherein one molecule e.g. A is ranked both higher and lower thananother molecule e.g. B.

Those skilled in the art will further recognize that PPOs may berepresented using permutations of molecules, or sets of permutations ofmolecules. Further, when represented using sets of permutations ofmolecules, the permutations in the set may be assigned weights such thata weighted PPO is represented as a probability distribution overpermutations of the molecules. Those skilled in the art will furtherrecognize that the set of all permutations forms the symmetric group.They will further recognize that cosets of the symmetric group representsets of partially constrained permutations of the molecules i.e. therank order of some molecules is specified; however, it is not specifiedfor all sets of molecules. Those skilled in the art will recognize,therefore, that PPOs may be represented as cosets of the symmetric groupand probability distributions over the cosets of the symmetric group(see “Cranking: Combining Rankings Using Conditional Probability Modelson Permutations”, Lebanon and Lafferty, Advances in Neural InformationProcessing Systems 15 incorporated herein in its entirety). Thoseskilled in the art will further recognize that a PPO may be representedas a cross product between a pair of sets. Given two sets {A,B,C,D} and{E,F,G}, the cross product consists of all pairs where the first elementis chosen from the first set and the second element is chosen from thesecond set. Furthermore, a PPO may be represented as a set of such crossproducts. Although the discussion below is written in terms of PPOs, andin particular it is written in terms of PPOs represented as pairs ofmolecules, those skilled in the art will recognize that alternativerepresentations, including those just described, are envisioned and arethus encompassed by the invention.

Further, this description refers to embodiments of the invention. Theinvention, however, is not limited to any specifically describedembodiments; rather, any combination of the described features, whetherrelated to a described embodiment, implements the invention. Further,although various embodiments of the invention may provide advantagesover the prior art, whether a given embodiment achieves a particularadvantage, does not limit the invention. Thus, the features,embodiments, and advantages described herein are illustrative and shouldnot be considered elements or limitations, except those explicitlyrecited in a claim. Similarly, references to “the invention” shouldneither be construed as a generalization of the inventive subject matterdisclosed herein nor considered an element or limitation of theinvention.

Creating a Pseudo-Partial Ordering (PPO) from Reported Measurements

(a) Continuous Measurements of a Property of Interest

In one embodiment, available measurements for the property of interestare used to create a PPO that includes a plurality of ranked pairs. Eachpair includes two molecules, wherein one molecule has a greater measuredvalue for the property of interest than the other molecule in the pair,e.g., the pair (A, B). Individual ranked pairs that satisfy any providedselection criteria are then included in a PPO. The ranked pairs of thePPO may then be used as training examples to train a molecularproperties model. Continuous measurements of the property of interest,relative to individual molecules, are used to select pairs of moleculesto include in a PPO. The measurements may be based on the results ofdirect experimentation, obtained from scientific literature, or on theresults of in-silico calculations generated using a software applicationconfigured to simulate chemical activity and reactions.

Similarly, a ranked pair may be constructed using different measurementsfor different substituent parts of a single molecule e.g. atoms or bondsin the molecule or different representations of a molecule e.g.alternative three dimensional conformations of the molecule.

FIG. 2 illustrates reported measurements of molecule activity that maybe used to construct a PPO, according to one embodiment of theinvention. In this example, individual molecules are represented usingthe capital letters A-F. Illustratively, graph 200 includes two sets ofmeasurements, 202 and 204, and graph 205 includes one set ofmeasurements 206. The measurements of molecule activity are plottedagainst the y axis of the graphs 200 and 205. Set 202 includes areported measurement for molecules A, B, D, and E. Set 204 includes areported measurement for molecules A, B, C, D, and F, at a differentconcentration level from set 202 (as plotted on the x axis). Set 205includes measurements reported for molecules C, B, and E. In additionset 208 includes activity measurements for molecules E and F reported inscientific literature (e.g., a peer-reviewed journal).

The measurements for an individual molecule plotted in graphs 200 and205 fluctuate. For example, the measurement for molecule B is differentin sets 202 204 and 206. When comparing data obtained from actuallaboratories, this state is common as different labs may employdifferent protocols or different quality standards. Also, theexperiments themselves may be carried out under substantially differentconditions. Thus, the reported value for an individual molecule may bedifferent, depending on the source of the measurement data. Further,when using measurements obtained in-silico (e.g., using a computersimulation), the measurements may also be inconsistent with thoseobtained in the laboratory e.g. the measurements may be obtained withrespect to arbitrary units or may be consistently biased higher or lowerthan reality.

The relative ordering of the molecules illustrated in graphs 200 and205, however, is fairly consistent, regardless of the source. This alsocommonly occurs when comparing actual data for the same set ofmolecules. Illustratively, molecule A is reported as more active thanany other molecule in each of the sets 202 and 204. The PPO of rankedpairs captures the relative nature of these measurements by representingmolecule activity as ranked data.

In one embodiment, the molecules that have reported measurements for theproperty of interest are used to generate a set of candidate pairs. FIG.2 illustrates the molecules A, B, C, D, E, and F, divided intosubdivisions 220, 222 and 224. The subdivision 220 includes candidatepairs 210 taken from set 202. Similarly, subdivisions 222 and 224include PPO candidate pairs 212 and 224 taken from sets 204 and 206,respectively.

Candidate pairs are assigned to subdivisions based on attributes of theproperty data. For example, all molecules tested under identicalconditions could constitute a subdivision. As another example, allmolecules tested against human enzyme could constitute a subdivision orall molecules for which Ki data are available might constitute asubdivision. Also, molecules may belong to several subdivisions. Forexample, subdivision 226 includes PPO candidate pairs from the union ofthe sets 202, 204 and 206.

For each subdivision, a set of criteria is used to select molecules toinclude in a ranked pair. The criteria used to determine a rankingbetween two molecules from the same subdivision may include, withoutlimitation, the relative magnitude of the measurement being above somethreshold, the absolute difference in magnitude of the reportedmeasurement being above some threshold, and the probability that themeasured values fall outside any experimental error intervals betweentwo molecules. For example, it may be known (or believed) thatlaboratory 1 has lower measurement uncertainty than laboratory 2, thusthe criteria for laboratory 2 will be more stringent. The appropriatecriteria are determined by considering any appropriate factorsincluding: the reported measurement uncertainty of an experiment, thereported measurement uncertainty of related experiments, measurementdifferences across species, measurement differences across laboratories,estimates for the error inherent in experimental data, uncertaintymeasurements regarding simulations carried out using computer software,and estimates or beliefs about any of these.

Whenever two molecules, e.g., molecule A and molecule B from set 202,belong to the same sub-division, and also satisfy the appropriatecriteria to be assigned a ranking (i.e., (A, B) or (B, A)) the orderedpair is added to a PPO. A pseudo-partial ordering is constructed bycombining all the ordered pairs that satisfy the criteria from eachsub-division of molecules.

From the set of molecules 202, the candidate pairs 210 include allpossible molecule rankings based on the reported values. Depending onthe selection criteria, however, not all possible pairs will be includedin the PPO. Illustratively, the close values of the reportedmeasurements of molecules (A, B) and (D, E) from set 202 may excludethese two pairs from the PPO. The ranked pairs (A, D), (A, E), (B, D),and (B, E), however, may satisfy the selection criteria and are includedin the pseudo partial ordering. Similarly, subdivision 222 and 224include candidate pairs 212 and 214. Note, one of the included rankedpairs, (C, B), from subdivision 224 ranks the same two moleculesdifferently than a ranked pair (B, C) from set 202. Because measurementsfor the property of interest may be obtained from different sources,different results may occur. Depending on the criteria used to selectcandidates from each subdivision, either or both of these ordered pairsmay be included in a PPO. Additional ranked pairs may be derived fromliterature values (e.g., the ranked pair (E, F)) from set 208.

Those skilled in the art will recognize that the measurements,orderings, candidate sets and PPOs illustrated in FIG. 2 are exemplary,and not meant to reflect the activity measurements for molecules thatwould be obtained using reported, estimated, or simulated values foractual molecules.

As discussed previously, the embodiments of the invention may be used tomodel molecular properties of individual atoms or bonds in a molecule,or of alternative representations of a molecule. In these cases thesubdivisions used for training data will typically consist of all of theatoms, bonds or representations of a given molecule.

(b) Discrete Measurements of a Property of Interest

In one embodiment, a PPO of ranked pairs may be constructed fromreported measurements that assign individual molecules with a discretelabel for the property of interest. For example, a molecule may belabeled as “active” or “inactive” for a given property of interest, or“positive” or “negative” for the property. Generally, a molecule labeledas “active” or “positive” may be paired with those labeled “inactive” or“negative” to form a ranked pair. The measurements for an individualmolecule may be obtained from any of the sources described aboveregarding continuous measurements of molecule activity.

Similarly, a ranked pair may be constructed using different measurementsfor different substituent parts or representations of a single moleculewhen one substituent is labeled “positive” and another substituent islabeled “negative” for a given property of interest e.g. the lability ofbonds may be analyzed where labile bonds may be labeled “positive” forlability and non-labile bonds may be labeled “negative”.

Often, the label assigned to an individual molecule is based on whetherthe measurement of a property of interest is above or below anarbitrarily selected threshold. For example, from data taken from Table1 of the Journal of Medicinal Chemistry, volume 48, pages 3114-3117, onecould choose an arbitrary cutoff of 10 micromolar and label moleculesthat bind to the TRalpha receptor as “positive” if their bindingaffinity is less or equal to 10 micromolar, and “negative” otherwise. Inthis case, compounds 4b, 2a, 3, 9e-k would be labeled “positive” andcompounds 9a-d “negative”. FIG. 3 illustrates graphs 300 and 309plotting reported measurements for four sets of molecules (sets 304,306, 310, and 312). In this example, individual molecules arerepresented using the capital letters A-G. Graph 300 includes two setsof measurements, 304 and 306, and graph 309 includes sets 310 and 312.The measurements of molecule activity are plotted against the y axis ofthe graphs 300 and 309.

Illustratively, graph 300 includes threshold 308 separating themolecules in sets 304 and 306 into two groups. Molecules above thethreshold are labeled “positive” for the property of interest, andmolecules below the threshold are labeled “negative.” Box 320illustrates molecules from graph 300 sorted based on whether themeasurement for a given molecule is above or below the threshold 308.From these sorted molecules, a PPO of ranked pairs may be generated byselecting each possible combination of a molecule selected from thoselabeled “active” paired with a molecule from those labeled “negative.”Note that this corresponds to the cross-product representation for PPOsdiscussed above and illustrated in 322. Additionally, ranked pairsconstructed in this manner may also be filtered using any appropriateselection criteria.

Similarly, a PPO may be constructed using the molecules plotted in graph309. The threshold 318 illustrated in graph 309, however, includes anupper bound 314 and a lower bound 316. This separation creates a region(illustrated using cross hatching) for which no assertion is maderegarding the property of interest. That is, molecules above the upperbound 314 are considered to be ranked above the molecules below thelower band 316. Molecules in the bounded region are not labeled eitherway, or used to construct a ranked pair. From these partitions, rankedpairs are constructed by combining molecules above the threshold withmolecules from below the threshold 318 as illustrated in 330. Using theupper and lower bounds (314 and 316) allows more stringent criteria tobe applied in selecting ranked pairs to include in PPO 324.

As described above, the ranked pairs included in a PPO may beconstructed using both continuous and discrete measurements of aproperty of interest. Additionally, ranked pairs may be created frommeasurements that directly report relative measurements of a property ofinterest for two (or more) molecules. For example, some experimentalprotocols may determine the relative activity of two molecules against atarget. Thus, if a measurement directly provides a ranking of twomolecules relative to a property of interest, then the two molecules maybe used to construct a ranked pair included in a PPO.

Virtual Molecules and Virtual Data

Optionally, ranked pairs may be generated using molecules for which ameasurement of the property of interest is unavailable for one moleculeincluded in the ranked pair. For example, the binding affinity of arandomly selected molecule against a protein receptor is likely to bevery low. Accordingly, a ranked pair may be created from such a moleculeand one known to have strong affinity for the protein receptor. Such aranking may be part of a PPO based on relative data measurements, or ona label indicating the molecule is above or below a given threshold. Ameasurement for the property of interest is assumed to be very low or“negative” relative to a molecule known to have a high level ofactivity, or labeled “positive.” Detailed examples of using assumedvalues for some activity measurements are described in a commonly ownedco-pending U.S. patent application Ser. No. 11/074,587 named aboveentitled “Methods for Molecular Property Modeling Using Virtual Data.”

Also, the molecules selected to include in a ranked pair may begenerated using computational simulation techniques. Methods forenumerating a set of synthesizable molecules are described in a commonlyowned U.S. Pat. No. 6,571,226, entitled “Method and Apparatus forAutomated Design of Chemical Synthesis Routes,” incorporated byreference herein in its entirety, alternative methods are possible andfall within the scope of this invention. The property data, for suchvirtual molecules may be generated based on reasonable assumptions, likethose regarding assumed virtual training data described in theapplication Ser. No. 11/074,587 or from software or hardwareapplications configured to simulate activity experiments to obtain ameasurement value. Illustrative embodiments of hardware and softwareconfigured to process molecular properties data are disclosed incommonly assigned U.S. patent application Ser. No. 10/449,948, “Methodand Apparatus For Quantum Mechanical Analysis of Molecular Systems,” andU.S. patent application Ser. No. 10/452,481 “Method and Apparatus forMolecular Mechanics Analysis of Molecular Systems.”

A PPO of ranked pairs may then be constructed using the virtualmolecules and/or virtual data using the techniques described above. Itis often the case that in silico simulations of molecular properties arefar more effective at producing rank orderings of molecules than theyare at predicting actual property values. In this case the output may beused to directly construct a PPO.

Weighting Data

In one embodiment weights may be assigned to the ranked pairs includedin a PPO. The value is assigned to reflect a measure of confidence inthe accuracy of a ranked pair. That is, the weighted value reflects anestimate of confidence in the validity of the assertion that molecule Ais ranked greater than molecule B relative to the property of interestfor the ranked pair, (A, B).

Additionally, molecules may be weighted to normalize the impact on thelearning process that can occur when one molecule appears over and overagain in the ranked pairs of a PPO. For example, a molecule with a highactivity value may appear in a disproportionate number of ranked pairsin a PPO. Multiple appearances of a molecule may bias the modelconstructed with such a PPO by exaggerating the importance of thefrequently occurring molecule. Another way in which a molecule mayappear a disproportionate number of times is in articles in thescientific literature. These articles commonly compare the activity ofnovel molecules against a common reference molecule. In this case therewill be a large number of reported data points for the referencemolecule. Once the completed training set is used to train a molecularproperties model, if the model “sees” the one molecule over and overagain as a learning example, it may simply learn to predict whether anarbitrary molecule is, in fact, the same as the one seen over and over.

Decreasing the weight assigned to each instance of a ranked pair forsuch a dominant molecule helps prevent this problem. For example, if thedominant molecule appears 10 times more frequently than others, eachinstance of a ranked pair with the dominant molecule may be weighted tocontribute a 1/10^(th) weight. Note however, this weighting is not areduction of confidence in the ranking of the dominant molecule; ratherit normalizes the contribution made by the dominant molecule.

Persons skilled in the art will recognize that embodiments of theinvention may use other techniques for assigning a weighted value to theranked pairs of a PPO. Accordingly, the weighting methods describedabove are included for illustrative purposes, and should not beconstrued to limit the scope of the invention.

FIG. 4 illustrates a method 400 for generating a PPO, according to oneembodiment of the invention. The method 400 begins at step 402 wheremeasurement values reported for a set of molecules are obtained. Themeasurement values assign a “score” to each molecule, relative to theproperty of interest. At step 404, the molecules are divided intosubdivisions of candidate ranked pairs (e.g., the subdivisions ofcandidate rankings 220 and 222 illustrated in FIG. 2).

At step 406, a loop comprising steps 408-414 is performed for eachsubdivision generated at step 404. At step 408, ranking criteria used toselect ranked pairs from the candidates of the current subdivision areselected. At step 410, each candidate pair in a given subdivision isprocessed to determine whether to include the candidate pair in the PPO.If the current candidate pair satisfies the selection criteria, then itis added to the PPO (step 414) and the next candidate pair is processed.Otherwise, the next candidate pair is processed. Steps 410-414 repeatfor each candidate pair in the current subdivision. Once allsubdivisions of molecules have been processed according to steps406-414, the method 400 proceeds to step 416.

At step 416, molecule activity data that assigns a label indicatingpresence or absence of a property of interest to each of a set ofmolecules is obtained. For example, each molecule may be labeled with anindication of “positive” or “negative.” At step 418, a loop comprisingsteps 418-424 tests pairs of molecules selected from those labeled atstep 416. Each candidate pair is evaluated (step 420). The evaluation ofstep 420 determines if the labels for the two molecules in a candidatepair indicate that one molecule is ranked above the other. If not, thenthe candidate pair is not added to the PPO. Otherwise, the candidatepair is added to the PPO. After evaluating the candidate pairs, themethod 400 proceeds to step 426. At step 426, data that directlyprovides a ranked ordering of two molecules relative to the property ofinterest is included in the PPO.

At step 428, the ranked pairs added to the PPO at steps 414, 424 and 426are merged, and the resulting PPO is output at step 430. Those skilledin the art will recognize that in a particular embodiment, not all typesof molecule data, as represented by steps 402, 416 and 426 are requiredto construct a PPO of ranked pairs. For example, in one embodiment, onlydata assigning a label of “positive” or “negative” to individualmolecules is used to construct the PPO. In another, only reportedmeasurements are used. In still another embodiment, a PPO may begenerated from ranked pairs generated from virtual data and virtualmolecules. The actual selection will depend on, among other factors, theavailability, cost, and reliability of data regarding the property ofinterest, and available computing power. Optionally, at step 429, theranked pairs selected to be included in the PPO may be weighted. Forexample, the ranked pairs may be weighted to normalize the impact of amolecule that occurs in multiple ranked pairs of the PPO.

While the foregoing was discussed in the context of molecular propertiesof a molecule as a whole, the invention equally may be applied to partsof a molecule e.g. atoms or bonds, or to alternative representations ofa molecule e.g. three dimensional conformations.

Training a Molecular Properties Model Using Ranked Data

As described above, embodiments of the invention may use severaldifferent techniques for selecting the ranked pairs to include in a PPO.Once the PPO is selected, the ranked pairs included in the PPO may beused as training examples to train a molecular properties model. Bothnovel machine learning algorithms, as well as general or specificmachine learning algorithms may use the ranked pairs included in the PPOas training examples. In one embodiment, the molecular properties modelincludes a software application configured to execute a machine learningalgorithm, using the ranked pairs of the PPO as training examples.Additionally, embodiments of the invention provide methods for usingnon-ranking algorithms (e.g., a classification or concept learningalgorithm) trained using a modified form of the ranked pairs included inthe PPO. Embodiments of the invention may use PPO data represented aspermutations, sets of permutations, cross products or sets of crossproducts as discussed previously. Several illustrative examples oflearning algorithms are described below.

FIG. 5 illustrates a method 500 for training a molecular propertiesmodel from a PPO of ranked pairs, according to one embodiment of theinvention. As described above, the PPO includes a set of ranked pairs,wherein each ranked pair orders the two molecules represented by thepair, relative to the property of interest.

The method 500 begins at step 502 by obtaining a set of moleculedescriptions together with measurements of the property of interest foreach molecule. Data regarding the property of interest may be in any ofthe forms described above (e.g., continuous measurements of activity ordiscrete labels), and further, molecules and property data may beobtained from the results of either actual or in-silico experimentation.At step 504, a PPO of the molecules is generated or obtained. Oneembodiment of a method for creating a PPO is illustrated by the method400 of FIG. 4.

Once the PPO of ranked pairs is constructed, a transformation process(step 506) is used to create a representation of the molecules in thePPO used to train a molecular properties model. In one embodiment, thetransformation process may include a software application configured toreceive a representation of the molecules in a ranked pair and generatea representation appropriate for a selected machine learning algorithm.For example, the transformation process may provide a vectorrepresentation of the molecules in a ranked pair, or may provide aconformational analysis of the molecules to generate a representationthat describes three dimensional conformations of the molecules in thepair. Embodiments of present invention may make use of representationsinvolving 10s to 10s of millions of features such as n-pointpharmacophores where n is 3, 4, 5 or larger.

Generally, the molecule descriptions generated by the transformationprocess at step 506 encode the structure, features and properties thatmay account for one molecule in a ranked pair having a greater activitythan the other molecule. Accordingly, properties such as presentfunctional groups, steric properties, electron density and distributionacross a functional group or across the molecule, atoms, bonds,locations of bonds and other chemical or physical properties of themolecule may all be used as part of the representation generated at step506.

When the present invention is applied to the modeling of molecularproperties of atoms or bonds in a molecule the representations maybe bedifferent. For example, a given atom may be represented by a list of allthe functional groups in which it is contained, or by a list of allpaths through the molecule in which it is contained. Similarly, when theinvention is applied to the modeling of alternative representations orconformations of a molecule the representation used by the learningalgorithm will contain features that differentiate between differentconformations.

At step 508, the molecule descriptions, together with the pseudo partialordering, are processed by a machine learning algorithm configured to“learn” using training examples that include the ranked pairs of a PPO.At step 510, the resulting molecular properties model is output. Theresulting molecular properties model is configured to generate aprediction for representations of molecules supplied to the model. Theprediction may be a prediction of a value for the property of interestfor a particular molecule, or may be a rank ordering (e.g., a PPO) for agroup of molecules supplied to the model. In a particular embodiment,the prediction provides a ranking for a pair of molecules, relative tothe property of interest.

The PPO used at step 508 may be represented as a list of pairs, or maybe represented as a list or set of permutations or a list or set ofcross products. Those skilled in the art will recognize that the PPOused in step 508 may be represented in many different ways. The presentinvention is not limited to any particular representation.

Illustrative Machine Learning Algorithms

In one embodiment, the pseudo partial ordering of ranked pairs issupplied to a learning algorithm not directly capable of using rankeddata at step 510, e.g., a classification learning algorithm. In such anembodiment, the ranked pairs included in the PPO are used as separatedata points and modified to include the label +1 if molecule A is rankedabove molecule B (e.g., (A>B)) or labeled −1 if molecule B is rankedabove molecule A (e.g., (A<B)). The resulting data set is fed to anarbitrary classification learning algorithm. Such an embodiment allowsclassification algorithms to use ranked data.

In another embodiment, a PPO is constructed and provided to a margin orkernel based learning algorithm at step 510. Each pair of molecules (A,B) is provided to the algorithm as (A−B) (i.e., an appropriaterepresentation of the difference between molecules A and B), and labeledas described above for classification algorithms. Such an embodiment maythen generate a linear combination of data points i.e. a model whosevalue on a new molecule C is a linear combination of the dot productsbetween representations of C and molecules in the training set. Thislinear combination can be interpreted as a linear combination ofmolecules and then used to assign a numerical score to arbitrarymolecules. The resulting model can be used to assign a total linearordering (or a partial ordering) to an arbitrary set of molecules.

In another embodiment, the learning algorithm used at step 508 maycomprise learning algorithms such as Boosting, a variant of Boosting,Rank Boosting, Alternating Decision Trees, Support Vector Machines, thePerceptron algorithm, Winnow, the Hedge Algorithm, an algorithmconstructing a linear combination of features or data points, DecisionTrees, Neural Networks, Genetic Algorithms, Genetic Programming,logistic regression, Bayes nets, log linear models, Perceptron-likealgorithms, Gaussian processes, Bayesian techniques, probabilisticmodeling techniques, regression trees, ranking algorithms, KernelMethods, Margin based algorithms, or linear, quadratic, convex, conic orsemi-definite programming techniques or any modifications orcombinations of the foregoing. Further, embodiments of the presentinvention contemplate using machine learning algorithms developed in thefuture, including newly developed algorithms or modifications of theabove listed learning algorithms.

In another embodiment, the learning algorithm used at step 508 attemptsto minimize (directly or indirectly) the area above a receiver operatorcharacteristic (ROC) curve (see “Model Selection via the AUC”, SaharonRosset, Proceedings of the 21^(st) International Conference on MachineLearning, 2004, incorporated herein in its entirety) constructed eitheron the training data or on an arbitrary set of molecules real, imaginedor virtual. The use of ROC curves allows the molecular properties modeloutput at step 510 to balance trade-offs between false positive andfalse negative test results as part of the learning process.

In another embodiment of the invention, the learning algorithm is anarbitrary algorithm that attempts to minimize (directly or indirectly)any cost function that relates to predictions made by the modelregarding the relative ordering of molecules. Those skilled in the artwill recognize that both currently known and novel learning algorithmsconfigured to process training examples in the form of a PPO of rankedpairs may be used at step 508, and are contemplated by the invention.

FIG. 6 illustrates a block diagram of data flow through a molecularproperties model 606, configured to generate a prediction for anarbitrary molecule, according to one embodiment of the invention. Theprediction 607 may provide a predicted measurement value for theproperty of interest, or may assign a label such as “active” or“inactive” to the molecule. Alternatively, in one embodiment, theprediction 607 may predict a PPO of ranked pairs for the moleculesprovided to molecular properties model 606.

Illustratively, the block diagram 600 shows input molecules 602, datapreprocessor 605, molecular properties model 606, and predictions 607.In one embodiment, preprocessor 605 constructs a representation of eachmolecule for which a prediction 607 is desired. For example, thetransformation process used to create molecule descriptions as part ofstep 506 from FIG. 5 may be used. The transformed representations arethen provided to molecular properties model 606. The molecularproperties model 606 then outputs a prediction 607 for the inputmolecules 602.

Embodiments of the present invention may make use of training data thatis not in a ranked form. In particular, embodiments of the presentinvention may make use of data that is not represented as a PPO.Further, embodiments of the invention may construct molecular propertiesmodels by optimizing a loss function that considers the relativeordering of the molecules in the training data. For example, embodimentsof the invention may use training data that represents molecules asbeing either active or inactive for a property of interest, or mayconstruct a molecular properties model by optimizing a function of therank order assigned to the molecules. An example of such a function isthe area above (below) the ROC curve. Similarly, embodiments of theinvention may use training data that represents the molecular propertyof interest as a continuous value. Such embodiments attempt to optimizea loss function of the rank order assigned to said molecules. Such aloss function penalizes incorrectly ordered molecules. Those skilled inthe art will recognize that learning algorithms that optimize a lossfunction of the rank order of a set of molecules (atoms or bonds) areimplicitly considering the training data as a PPO.

Embodiments of the present invention may be used to construct molecularproperties models when the training data is biased, or when the optimaltrade-off between false positives and false negatives is unknown apriori. The invention constructs a ranking model by generating rankingdata (e.g. a PPO) or by optimizing a function of the rank ordering ofmolecules in the training set. Subsequent to model construction aclassification model may be obtained by determining a threshold value orcutoff molecule. Molecules that score above the threshold, or rank abovethe cutoff molecule are considered in one class, the remaining moleculesare considered in the other class. The threshold value or cutoffmolecule may be determined a posteriori based on information thatbecomes available, e.g., a specification of the optimal trade-offbetween false positives and false negatives.

Molecules predicted to exhibit the property of interest, predicted tohave a high measurement value for the property of interest, or otherwiseidentified by molecular properties models constructed by the presentinvention, may be identified for further investigation, includingexperimentation carried out in the laboratory or using additionalcomputer simulation techniques. Given the current availability of datatransport mechanisms, predictions generated for a test molecule at onelocation may be transported to other locations using well known datastorage and transmission techniques. And predictions may be verifiedexperimentally at the other locations. For example, a computer systemmay be located in one country and configured to generate predictionsabout the property of interest for a selected group of molecules, thisdata may be then be transported (or transmitted) to another location, oreven another country, where it may be the subject of furtherinvestigation, e.g., laboratory confirmation of the prediction orfurther computer-based simulations.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1-10. (canceled)
 11. A method for training a molecular properties model, comprising: obtaining a pseudo-partial ordering of molecules, wherein the pseudo partial ordering includes at least a representation of a first and second molecule, ordered relative to one another and a property of interest; and generating a representation of the molecules included in the pseudo partial ordering of molecules that is appropriate for a selected machine learning algorithm, wherein the pseudo partial ordering of molecules is provided to the selected machine learning algorithm, and wherein executing the selected machine learning algorithm, using the pseudo partial ordering, trains a molecular properties model configured to generate a prediction regarding additional molecules supplied to the model.
 12. The method of claim 11, wherein the molecular properties model generates predictions related to a property of interest selected from at least one of a pharmacokinetic property, pharmacodynamic property, physiological or pharmacological activity, toxicity or selectivity; a chemical property including reactivity, binding affinity, pKa, or a property of a specific atom or bond in a molecule; or a physical property including melting point, solubility, a membrane permeability, and a force-field parameter.
 13. The method of claim 11, wherein the selected machine learning algorithm comprises a classification learning algorithm.
 14. The method of claim 11, wherein the selected machine learning algorithm comprises a kernel based learning algorithm.
 15. The method of claim 11, wherein the selected machine learning algorithm comprises a variant of a Boosting algorithm, RankBoost algorithm, Alternating Decision Trees algorithm, Support Vector Machines algorithm, a Perceptron algorithm, Winnow, a Hedge Algorithm, decision trees, neural networks, genetic algorithms, genetic programming or any modifications thereof modified to process the pseudo partial ordering of ranked pairs.
 16. The method of claim 11, wherein the selected machine learning algorithm is configured to minimize, either directly or indirectly, an area above, or below, a receiver operator characteristic curve.
 17. The method of claim 11, wherein the selected machine learning algorithm is configured to minimize, either directly or indirectly, a function of the rank ordering of molecules in the pseudo partial ordering.
 18. The method of claim 11, further comprising, determining an accuracy of the prediction for the additional molecule by carrying out laboratory experimentation using physically existing samples of the additional molecule, or by performing a research study using physical samples of the additional molecule.
 19. The method of claim 11, wherein generating the representation of the molecules included in the pseudo partial ordering of molecules comprises: generating a vector representation of the molecules, wherein the vector representation is configured to encode the structure of the molecules included the pseudo-partial ordering; or comprises generating an n-point pharmacophore representation of the molecules included in the pseudo-partial ordering.
 20. The method of claim 11, wherein a threshold value or cutoff molecule is selected for the molecular proprieties model and used to create a classification model.
 21. The method of claim 11, wherein the at least one additional molecule comprises two or more additional molecules, and wherein the prediction comprises a ranked ordering of the two or more additional molecules, relative to one another and to the property of interest.
 22. The method of claim 11, wherein at least two of the molecules in the training set are alternative representations of the same physical molecule and the property of interest is a property of the alternative representations.
 23. The method of claim 11, wherein at least two of the molecules in the training set are encoded to represent different atoms, bonds or substituent groups of the same molecule and the property of interest is a property of the atoms, bonds or substituent groups. 24-34. (canceled)
 35. A computer-storage medium containing a program which, when executed by a processor, performs a method for training a molecular properties model, comprising: receiving a pseudo-partial ordering of molecules, wherein the pseudo partial ordering includes at least a representation of a first and second molecule, ordered relative to one another and a property of interest; and generating a representation of the molecules included in the pseudo partial ordering of molecules that is appropriate for a selected machine learning algorithm, wherein the pseudo partial ordering of molecules is provided to the selected machine learning algorithm, and wherein executing the selected machine learning algorithm, using the pseudo partial ordering, trains a molecular properties model configured to generate a prediction regarding additional molecules supplied to the model. 36-40. (canceled) 